\chapter{Database Implementation}
This chapter provides a description of the ontology model used for creating hierarchical structure of thesis domain. It will explain how to connect and work with the Virtuoso database. Then it will be clarified how the extracted information will be converted into a graph and stored in the database.

\section{Ontology}
This section is a recapitulation of the theoretical section Modeling the information\ref{sec:Ontology}. Informations is divided into classes:
\begin{itemize}
	\item \emph{Terms} - Contains information about subjects and objects, its URIs, lemmas and textual representations
	\item \emph{Predicates} - Contains information about predicates, its URIs, lemmas and textual representations
	\item \emph{Terminologies} - Contains information about terminologies, its URIs, lemmas, textual representations, overall frequencies, frequencies, number of documents in that occur, TF-IDF
	\item \emph{Document} - Contains document name, path to the original document, sentences, terms, predicates, terminologies count, triples and terminologies
\end{itemize}

The ontology used in the thesis can be found on the included CD. During a design of an ontology, how to hierarchically structure information is focused. What is relevant for the document and what is common for all documents. Resource (term, predicate, terminology) instance is relevant only for a document and it denotes that the document contains that kind of information. The textual representation is not relevant for a document because it denotes only instance of a particular resource. Therefore it does not need to be in a document graph and the textual representations can be grouped in one common graph. Graphs that has well known URI's, textual representations and lemma of the resources has to be created. Using URI's prefix \emph{semjobKB:}\url{http://datlowe.org/semjobKB/data/document/} the graphs are:

\begin{itemize}
	\item{semjobKB:term\#} for subjects and objects, graph contains individual textual representations of the resources and their lemmas
	\item{semjobKB:predicate\#} for predicates, graph contains individual textual representations of the resources and its lemmas
	\item{semjobKB:Terminology\#} for terminologies, graph contains individual textual representations of the resources, its lemmas, overall frequencies, number of documents in that occur
\end{itemize}

To optimize search based on a textual representation of some resources, it do not need to be gone trough all documents. It suffice to look into these graphs, get the resource URIs having the same textual representations and search for documents based these URIs. This ontology allows us to efficiently store the informations and provide fast search services.

\section{Working with Virtuoso database}
A research has been done to find \emph{JAVA API} that would allow to connect, update, store and retrieve data from within a Java application. That functionality is available in the open source Semantic framework called \emph{Jena}\cite{Jena}. \emph{Jena's} subproject called \emph{virt-jena}\cite{JenaApi} is especially designed to work with Virtuoso database and therefore this library is used in the application. To operate with the application only the database \emph{url} location, user login and password that are injected into the application in \emph{config.properties} file has to be set. The library enables to create Virtuoso graphs, fill them with triples and store them. The \emph{virt-jena} does not have any tutorials only \emph{Javadoc} documentation and few example test that are part of the project.

\section{Converting and storing extracted informations into a graph}
The previous version of the thesis directly stored the graph into the application. The application qualitative results of the IE showed that some of the triples and terminologies contain invalid data, will be discussed at the end of this thesis. To avoid adding unrelated or unwanted data a step was added. This new step will allow user to select what he thinks is best suited to describe the document or at least filter the founded triples and terminologies.

\subsection{Selecting extracted knowledge to store}
Extracted knowledge is displayed to a user prior its actual persistence in the database. The reason for that is to allow user to deselect irrelevant triples and terminologies. Just before the knowledge is displayed to the user, for each triple and terminology is searched in the database. Each terminology and triple then contains the information if they are stored already in the database or not. After confirming the selection only the selected knowledge will be passed to actual storing.

\subsection{Storing extracted knowledge into the database}
The extracted informations that has been filtered by a user in the previous step needs to be converted into the graph based on the described ontology. Extracted knowledge contains list of triples, and list of terminologies with their frequencies. The process of converting informations into the graph is:
\begin{enumerate}
	\item Check if the document is exists in the database, if yes then:		
	\begin{enumerate}			
		\item For each terminology URIs in the document subtracts the overall frequency by the frequency of the terminology and decrease by one the number of documents in which the terminology occurs
		\item Remove the graph from the database and create an empty one
	\end{enumerate}
	\item else:
	\begin{enumerate}			
		\item Create an empty graph
	\end{enumerate}
	\item For each terminology:
	\begin{enumerate}
		\item Escape all characters in the lemma that are not allowed in URI and create URI from the term prefix {semjobKB:Terminology\#} and this escaped lemma.
		\item Create new triple for terminology, predicate semjobKB:hasStringRepresentation and literal having the textual representation of the terminology
		\item Create new triple for terminology, predicate semjobKB:hasLemma and literal having the lemma		
		\item Crate or update the terminology overall frequency and number of occurrences
		\item Add the triples into the terminology graph				
		\item Create triple from terminology resource URI, has:Frequency predicate and literal filled with the frequency
		\item Add it into the document graph
	\end{enumerate}	
	\item For each triple:
	\begin{enumerate}
		\item Extract the subject and object from the triple
		\item Escape all characters in the lemma that are not allowed in URI and create URI from the term prefix {semjobKB:term\#} and this escaped lemma.
		\item Create new triple for subject and object, predicate semjobKB:hasStringRepresentation and literal having the textual representation of the subject or object
		\item Create new triple for subject and object, predicate semjobKB:hasLemma and literal having the lemma
		\item Add the triples into the term graph
		\item Make the same for predicate and add the textual representation triples into the predicate graph
		\item Create triple from subject, predicate, object resource URIs and add it into the document graph
	\end{enumerate}
	\item Create triples containing the sentences, terminology, terms, predicates count, language, document name, document path and store them into the graph.
	\item Method graph.close() commits the changes made and uploads the graph into the database
\end{enumerate}

This algorithm ensures that the same document can be saved again. The purpose behind it is that the use can have multiple IE search strategies and he would like to test which is serves better over the others. The algorithm ensures that terminology overall frequencies and the number of occurrences is recalculated when the document is removed and replaced with a updated one. The predicates used to create triples about document details and statistics are in the ontology and can be found in the appended CD.

\section{Implementation}
Implementation is done in module \emph{dbVirtuoso} in a class {DatabaseService} that implements \emph{IDatabaseService} and provides method \emph{(IDocumentDetail storeDocument(IExtractedKnowledge extractedKnowledge, File documentFile, ELanguage language))}. The parameters are: extracted knowledge, file containing the original document and the language of the document. The language has default value set for the Czech language, but in future improvement counted and other languages could be use. The method \emph{storeDocument} after successful creation and saving the graph returns the graph details to be displayed to a user in the application GUI. The functionality to remove a document from the database is not supported. However an administrator can remove graphs via Virtuoso administrator interface.